Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic).

I magine that you're a busy family physician and that you've found a rare free moment to scan the recent literature. Reviewing your preferred digest of abstracts, you notice a study comparing emergency physicians' interpretation of chest radiographs with radiologists' interpretations. 1 The article catches your eye because you have frequently found that your own reading of a radiograph differs from both the official radiologist reading and an unofficial reading by a different radiologist, and you've wondered about the extent of this disagreement and its implications. Looking at the abstract, you find that the authors have reported the extent of agreement using the κ statistic. You recall that κ stands for " kappa " and that you have encountered this measure of agreement before, but your grasp of its meaning remains tentative. You therefore choose to take a quick glance at the authors' conclusions as reported in the abstract and to defer downloading and reviewing the full text of the article. Practitioners, such as the family physician just described, may benefit from understanding measures of observer variability. For many studies in the medical literature, clinician readers will be interested in the extent of agreement among multiple observers. For example, do the investigators in a clinical study agree on the presence or absence of physical, radiographic or laboratory findings? Do investigators involved in a systematic overview agree on the validity of an article, or on whether the article should be included in the analysis? In perusing these types of studies, where investigators are interested in quantifying agreement, clinicians will often come across the kappa statistic. In this article we present tips aimed at helping clinical learners to use the concepts of kappa when applying diagnostic tests in practice. The tips presented here have been adapted from approaches developed by educators experienced in teaching evidence-based medicine skills to clini-cians. 2 A related article, intended for people who teach these concepts to clinicians, is available online at www. Defining the importance of kappa • Understand the difference between measuring agreement and measuring agreement beyond chance. • Understand the implications of different values of kappa. • Understand the basics of how the kappa score is calculated. • Understand the importance of " chance agreement " in estimating kappa. • Understand how to calculate the kappa score given different distributions of positive and negative results. • Understand that the more extreme the distributions of positive …